Glasses with closed captioning, voice recognition, volume of speech detection, and translation capabilities

ABSTRACT

The glasses with display may include a bridge, two temples hingedly coupled to the bridge, and a directional microphone array, the directional microphone array including two or more microphones positioned on the bridge or the temples. The glasses with display may also include a user microphone array, the user microphone array including one or more microphones positioned on the temples and oriented toward the mouth of a user wearing the glasses with display or one or more bone conduction microphones. In addition, the glasses with display include two lenses positioned in the bridge, at least one of the lenses including a display, the display visible by the user, the display including one or more of a directional display, closed caption display, and user volume display. The glasses with display additionally include a processor adapted to receive audio signals from the directional microphone array and the user microphone array, or from a separate mobile device, the processor adapted to control the display.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation application which claims priorityfrom U.S. utility application Ser. No. 16/704,256, filed Dec. 5, 2019,which is itself a nonprovisional application which claims priority fromU.S. provisional application No. 62/781,374, filed Dec. 18, 2018.

TECHNICAL FIELD/FIELD OF THE DISCLOSURE

The present disclosure relates to personal wearable electronic devices,and specifically to eyeglasses.

BACKGROUND OF THE DISCLOSURE

People with profound or total hearing loss are unable to converse easilywith others. They often are unable to hear some or all of the words thatare spoken to them, or the words they do hear are misunderstood. Theinability to communicate well can occur in many different settings,including, but not limited to, one-on-one meetings, group meetings andphone calls. In addition, people with profound or total hearing loss areunable to hear spoken words in places of worship, movie theaters, livetheaters, and other group settings where one or several persons arespeaking to a larger group of observers.

People with profound hearing loss are often unable to hear their ownvoices unless they speak very loudly. Other people with total hearingloss are unable to hear how loudly they are speaking. Often, withoutknowing how loudly they are speaking, these persons speak at volumesthat are much louder than necessary in a particular setting.

Some persons who are affected by profound or total hearing loss areproficient at lip reading, and many others are unable to develop theability to lip read.

People, regardless of their ability to hear well, are also often facedwith the task of communicating with others even though neither speaksthe other's language. Such encounters may occur in business, travel, anddiplomacy, to name a few such instances.

SUMMARY

Eyeglasses with display are disclosed. The glasses with display mayinclude a bridge, two temples hingedly coupled to the bridge, and adirectional microphone array, the directional microphone array includingtwo or more microphones positioned on the bridge or the temples. Theglasses with display may also include a user microphone array, the usermicrophone array including one or more microphones positioned on thetemples and oriented toward the mouth of a user wearing the glasses withdisplay or one or more bone conduction microphones. In addition, theglasses with display include two lenses positioned in the bridge, atleast one of the lenses including a display, the display visible by theuser, the display including one or more of a directional display, closedcaption display, and user volume display. The glasses with displayadditionally include a processor adapted to receive audio signals fromthe directional microphone array and the user microphone array, or froma separate mobile device, the processor adapted to control the display.

A method is disclosed. The method includes providing glasses withdisplay. The glasses with display include a bridge and two templeshingedly coupled to the bridge. The glasses with display further includea directional microphone array, the directional microphone arrayincluding two or more microphones positioned on the bridge or thetemples and two lenses positioned in the bridge, at least one of thelenses including a display, the display visible by the user. Inaddition, the glasses with display include a processor. The method alsoincludes receiving sound from a speaker with the directional microphonearray as an audio signal and recognizing speech from the audio signalwith the processor. The method further includes outputting text speechoutput from the processor and displaying the text speech output with thedisplay.

A method is disclosed. The method includes providing glasses withdisplay. The glasses with display include a bridge, two temples hingedlycoupled to the bridge, and a user microphone array, the user microphonearray including one or more microphones positioned on the temples andoriented toward the mouth of a user wearing the glasses with display, orone or more bone conduction microphones. The glasses with display alsoinclude two lenses positioned in the bridge, at least one of the lensesincluding a display, the display visible by the user and a processor.The method also includes receiving sound with the user microphone array,the sound corresponding to the sound of the user's voice, the soundoutput as an audio signal. In addition, the method includes analyzingthe volume of the audio signal with the processor, outputting uservolume information from the processor, and displaying the user volumeinformation with the display.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is best understood from the following detaileddescription when read with the accompanying figures. It is emphasizedthat, in accordance with the standard practice in the industry, variousfeatures are not drawn to scale. In fact, the dimensions of the variousfeatures may be arbitrarily increased or reduced for clarity ofdiscussion.

FIG. 1 depicts a perspective view of glasses with display consistentwith at least one embodiment of the present disclosure.

FIG. 2 depicts a schematic diagram of the glasses with display of FIG.1.

FIG. 3 depicts a representation of the view of a user of glasses withdisplay consistent with at least one embodiment of the presentdisclosure.

FIG. 3A depicts an alternative representation of the view of a user ofglasses with display consistent with at least one embodiment of thepresent disclosure.

FIG. 3B depicts an alternative representation of the view of a user ofglasses with display consistent with at least one embodiment of thepresent disclosure.

FIG. 4 depicts a schematic diagram of a closed captioning operationconsistent with at least one embodiment of the present disclosure.

FIG. 5 depicts a schematic diagram of a translation operation consistentwith at least one embodiment of the present disclosure.

FIG. 6 depicts a schematic diagram of a volume detection operationconsistent with at least one embodiment of the present disclosure.

DETAILED DESCRIPTION

It is to be understood that the following disclosure provides manydifferent embodiments, or examples, for implementing different featuresof various embodiments. Specific examples of components and arrangementsare described below to simplify the present disclosure. These are, ofcourse, merely examples and are not intended to be limiting. Inaddition, the present disclosure may repeat reference numerals and/orletters in the various examples. This repetition is for the purpose ofsimplicity and clarity and does not in itself dictate a relationshipbetween the various embodiments and/or configurations discussed.

FIG. 1 depicts glasses with display 100. Glasses with display 100 mayinclude frame 101. Frame 101 may include bridge 103; arms or temples 105a, 105 b; and lenses 107. Temples 105 a, 105 b may be hingedly coupledto bridge 103. Lenses 107 may be positioned within openings formed inbridge 103. Lenses 107 may be prescription lenses, nonprescriptionlenses, tinted lenses, or may be formed from flat glass or othermaterial.

One or both of lenses 107 may include display 109 positioned to allowinformation to be displayed to a wearer of glasses with display 100 asfurther described below. In some embodiments, display 109 may begenerated by one or more micro-LED displays. In some embodiments,display 109 may be generated by projecting an image onto one or both oflenses 107 by, for example and without limitation, projector 111 asshown in FIG. 1. In some embodiments, projector 111 may be positioned intemple 105 a. In some embodiments, one or both of lenses 107 may includea filter or other layer or coating that allows display 109 to appear infocus to a user and to make display 109 appear to be further from theuser's eye than lens 107. In some embodiments, display 109 may be on oneof lenses 107, on both of lenses 107, or on an additional display modulewithin the user's field of view. In some embodiments in which display109 is on both of lenses 107, display 109 may be generated as athree-dimensional view or hologram. In some embodiments, display 109 maybe used to display one or more of a closed caption display anddirectional display that represent words spoken to a user of glasseswith display 100 and a user volume display that indicates the volume ofthe user's voice as further described below. Display 109 may be usedsimilarly for a phone call.

In some embodiments, glasses with display 100 may include directionalmicrophone array 113 adapted to receive sound from the environmentsurrounding a user of glasses with display 100 and may be used todetermine the direction of the incoming sound. In some embodiments,directional microphone array 113 may include one or more microphones,such as microphones 113 a-d depicted in FIG. 1, which may be positionedin bridge 103 (with another microphone positioned on the opposite side).In some embodiments, additional microphones of directional microphonearray 113 may be positioned in one or both of temples 105 a and 105 b.

In some embodiments, glasses with display 100 may include usermicrophone array 115 which may include one or more user microphones 115a, 115 b, 115 c. User microphones 115 a, 115 b, 115 c may be positionedto receive sound generated by the user of glasses with display 100, i.e.the user's own voice. In some embodiments, user microphones 115 a, 115 bmay be positioned in a lower side of temples 105 a, 105 b such that usermicrophones 115 a, 115 b point downward toward the user's mouth. In someembodiments, user microphones 115 a may be bone conduction microphones.In yet other embodiments, where a user microphone is a bone conductionmicrophone, the bone conduction microphone may be microphone 115 c, forexample.

In some embodiments, glasses with display 100 may include processormodule 117. Processor module 117 may include a processor as furtherdescribed below that is used to receive audio signals from directionalmicrophone array 113, user microphone array 115, and any other sensorsand to control display 109. In some embodiments, processor module 117may be positioned in one of temples 105 a, 105 b. In some embodiments,processor module 117 may be positioned in the same temple (105 a asdepicted in FIG. 1) as projector 111 or may be in the opposite temple(105 b as depicted in FIG. 1). In some embodiments, glasses with display100 may include power supply 119. Power supply 119 may include, forexample and without limitation, one or more batteries. In someembodiments, batteries may be user replaceable or may be rechargeable.In some embodiments, power supply 119 may, for example and withoutlimitation, include one or more lithium-ion, lithium-polymer,nickel-cadmium, nickel metal hydride, or alkaline batteries. In someembodiments, power supply 119 may include a receptacle for rechargingthe batteries such as, for example and without limitation, a USB jack orother suitable connector. In other embodiments, power supply 119 mayinclude a wireless charging system to connect to an external wirelesscharging pad.

FIG. 2 depicts a representative schematic diagram of glasses withdisplay 100. Directional microphone array 113 and user microphone array115 may be used to receive sound such as words spoken to a user ofglasses with display 100 or spoken by a user of glasses with display 100and may convert the sound to audio signals 121 a, 121 b, which may beinput to processor 123. Processor 123 may analyze audio signals 121 a,121 b as further discussed below, to determine one or more of what wordswere spoken to a user, the direction of the speaker of the words spokenrelative to the user, and the volume of the user's voice. In someembodiments, processor 123 may further translate the words spoken to theuser. Processor 123 may be operatively coupled to display 109 such thatprocessor 123 causes display 109 to generate and display one or more ofdirectional display 125, closed caption display 127, and user volumedisplay 129.

In some embodiments, glasses with display 100 may include sensor package130. Sensor package 130 may include one or more sensors including, forexample and without limitation, one or more accelerometers, gyros, andmagnetometers positioned to determine the direction that a user ofglasses with display 100 is looking and any change in orientation ofglasses with display 100. This directional and movement information maybe used by processor 123 as further described below.

FIGS. 3 and 3A depict examples of display 109, 109 a respectively aswould be seen by a user of glasses with display 100. These displays aremerely examples and are not meant to limit the scope of this disclosure.

In some embodiments, directional display 125 may indicate the directionrelative to the user of glasses with display 100 from which speech isdetected. In some embodiments, as shown in FIG. 3, directional display125 may include one or more visual directional indicators such as arrow131. In such an embodiment, arrow 131 may appear to rotate to pointtoward speaker 15 while speaker 15 is speaking to the user. In someembodiments, when a different speaker is speaking to the user, arrow 131may then point to the different speaker. In other embodiments, asdepicted in FIG. 3A, directional display 125 a may instead modify thepositioning of one or more elements of display 109 a such as, forexample and without limitation, closed caption display 127′ such thatclosed caption display 127′ moves to a position on display 109 a tocorrespond with the relative direction between the current direction ofview of the user of glasses with display 100 and speaker 15. In certainembodiments, such as depicted in FIG. 3B, text may appear under or inthe direction of the speaker, such that when there are two or morepeople speaking, such as speakers 15 a and 15 b, text will appear undereach such speaker. on closed caption display 127′, shown as text boxes133 a and 133 b, respectively. Multiple displays of text may thus appearsimultaneously if each other person has spoken.

In some embodiments, glasses with display 100 may include a camerapositioned on bridge 103. The camera may, for example and withoutlimitation, be used to determine the direction the user is facing. Insome embodiments, the camera may detect and analyze speaker 15. In someembodiments, glasses with display 100 may use information from sensorpackage 130 and the camera to determine the position of speaker 15within the user's field of view and may display one or more visualindicators to identify speaker 15 within display 109. For example, insome embodiments, display 109 may indicate speaker 15 within the user'sfield of view by positioning a box or other shape around the face ofspeaker 15 from the user's viewpoint. In some embodiments, glasses withdisplay 100 may include one or more sensors positioned to determine thedirection to which the eyes of the user are pointing, for example andwithout limitation, properly position elements of display 109 within theuser's field of view.

In some embodiments, closed caption display 127, 127′ may be shown asone or more lines of text in text box 133 displayed on display 109, 109a. Text box 133 may display text corresponding to what is said byspeaker 15 as further described herein below.

FIG. 4 depicts a schematic diagram of closed captioning operation 200consistent with at least one embodiment of the present disclosure.During operation of closed captioning operation 200, processor 123 mayreceive audio signal 121 a from directional microphone array 113.Processor 123 may process audio signal 121 a by recognizing speech atspeech recognition 201. Speech recognition 201 may analyze audio signal121 a and determine what words were spoken, converting audio signal 121a into text speech output 203. Text speech output 203 may be passed toclosed caption display 127 of display 109.

In some embodiments, processor 123 may further process audio signal 121a by determining the direction of the incoming speech at directionanalysis 205. Direction analysis 205 may analyze audio signal 121 a todetermine the direction from which the speech analyzed by speechrecognition 201 is received. Direction analysis 205 may use, for exampleand without limitation, differential analysis of the signals received byeach microphone of directional microphone array 113 to determine thedirection. Direction analysis 205 may output directional information207, which may be used by directional display 125 to indicate thedirection from which the speech analyzed by speech recognition 201 isreceived with display 109.

In some embodiments, direction analysis 205 may be used to determinewhen two or more speakers are detected and to determine which speech isdetected from which speaker. This directional information 207 may beused by speech recognition 201 to refine text speech output 203 to, forexample and without limitation, isolate the speech of a single speakerfor display, to output multiple text speech outputs 203 for multipleusers, or to otherwise affect speech recognition 201.

In some embodiments, the outputs 132 from one or more sensors of sensorpackage 130 may be used by direction analysis 205 to, for example andwithout limitation, recognize changes in orientation of glasses withdisplay 100, thereby allowing the directional model to be updated basedon the change in orientation of glasses with display 100. Without beingbound to theory, the change in orientation of glasses with display 100may be reflected by updating directional display 125 to indicate thatthe direction of the source of the speech analyzed by speech recognition201 relative to the field of view of the user of glasses with display100 has changed, and updating directional display 125 to correspond withthe change in orientation of glasses with display 100.

In some embodiments, speech recognition 201 may further analyze audiosignal 121 a to determine whether a single speaker or multiple speakersare detected. Speech recognition 201 may use, for example and withoutlimitation, frequency analysis, volume analysis, or other techniques todifferentiate the speech of multiple speakers and to determine a primaryor selected speaker to analyze. In some embodiments, speech recognition201 may store the results of this analysis to, for example and withoutlimitation, recognize speakers and, in some embodiments, identifyspeakers based on past analysis.

In some embodiments, speech recognition 201 may filter out the user'sown voice to prevent or reduce the likelihood that the user's own speechis displayed on closed caption display 127. In some such embodiments,speech recognition 201 may use, for example and without limitation, oneor more of frequency analysis, volume analysis, or differential analysisbetween audio signal 121 a from directional microphone array 113 andaudio signal 121 b from user microphone array 115 to differentiatespeech from the user and from a speaker. In some embodiments, speechrecognition 201 may filter out background noise from audio signal 121 afrom directional microphone array 113.

FIG. 5 depicts a schematic diagram of translation operation 300consistent with at least one embodiment of the present disclosure.During operation of translation operation 300, processor 123 may receiveaudio signal 121 a from directional microphone array 113. Processor 123may process audio signal 121 a by recognizing speech at speechrecognition 301 and may determine the direction of the incoming speechat direction analysis 305 as discussed herein above with respect toclosed captioning operation 200. In some embodiments, a user may input aselected language expected to be used by a speaker at language selection307. The selected language may be input into speech recognition 301 andused, for example, to select a language model to be used by speechrecognition 301 when generating text speech output 303. Text speechoutput 303 may be translated into a language understood by the user attranslation 309. Translation 309 may translate text speech output 303based on the user-selected language at language selection 307 togenerate translated text speech output 311. Translated text speechoutput 311 may be passed to closed caption display 127 of display 109,allowing the user of glasses with display 100 to understand the speechof a speaker speaking a different language by reading the text of closedcaption display 127. Thus, multiple speakers of a foreign language couldbe accommodated, as described hereinabove.

FIG. 6 depicts a schematic diagram of volume detection operation 400.During operation of volume detection operation 400, processor 123 mayreceive audio signal 121 b from user microphone array 115. Processor 123may process audio signal 121 b by determining the volume of the user'svoice at volume analysis 401. Volume analysis 401 may compare theamplitude of audio signal 121 b to one or more preset volume thresholdvalues, which may be preset or set by a user at volume thresholds 403.Volume analysis 401 may output user volume information 405 correspondingto the volume of the user's voice compared to the preset volumethreshold values, the ambient noise level of the current surroundings,or other scale. Volume information 405 may be passed to user volumedisplay 129 to indicate to a user of glasses with display 100 the volumeof the user's voice with display 109. In some embodiments, user volumedisplay 129 may provide a visual indication of the volume of the user'svoice to the user who may be otherwise unable to assess the volume dueto, for example and without limitation, hearing impairment.

In some embodiments, volume analysis 401 may process audio signal 121 bto ensure that only the user's voice is used to determine user volumeinformation 405. For example, in some embodiments, audio signal 121 afrom directional microphone array 113 may be used to identify sounds inthe environment other than the user's voice in order to distinguish theuser's voice for volume analysis 401.

In some embodiments, as depicted in FIG. 3, volume display 129 mayinclude volume graph 135. Volume graph 135 may visually display thevolume of a user's voice while the user is speaking. In someembodiments, volume graph 135 may include visual indicators of volumethresholds including, for example and without limitation, low-volumethreshold 137 and high-volume threshold 139. Volume graph 135 mayinclude indicator 141 to indicate the volume of the user's voicerelative to low-volume threshold 137 and high-volume threshold 139 orrelative to the scale of volume graph 135. In some embodiments, volumegraph 135 may be triangular or rectangular, or it may consist of aseries of dashes or other markings, to visually indicate which end ofvolume graph 135 indicates low-volume and which end indicateshigh-volume. In some embodiments, sections of volume graph 135 may becolor-coded to indicate when the user's voice is within normal volumerange 143, low-volume range 145, or high-volume range 147, specified bylow-volume threshold 137 and high-volume threshold 139. In someembodiments, for example and without limitation, low-volume range 145may be indicated by a yellow or white color, normal volume range 143 bya green or gray color, and high-volume range 147 by a red or blackcolor. In some embodiments, high-volume range 147 may include one ormore further indicators such as accentuation lines 149 to visuallyindicate that the user is speaking at a volume above the high-volumethreshold 139. In some embodiments, one or more elements of volume graph135 may otherwise indicate the volume of the user's voice by, forexample and without limitation, flashing, changing in color, orotherwise indicating information relevant to the user regarding thevolume of the user's voice.

In some embodiments as depicted in FIG. 2, glasses with display 100 mayinclude communications module 150. Communications module 150 may allowfor wired or wireless communication between glasses with display 100 andother, external devices. Communications module 150 may include, forexample and without limitation, one or more of a WI-FI transceiver,Bluetooth transceiver, or cellular transceiver. In some embodiments,glasses with display 100 may use communications module 150 to connect tomobile device 151 such as, for example and without limitation, asmartphone. Mobile device 151 may, in some embodiments, be used toconfigure settings of glasses with display 100 including, for exampleand without limitation, language selection 307 used for translationoperation 300, volume thresholds 403 used for volume detection operation400. In some embodiments, mobile device 151 may be used to configureparameters of display 109 including, for example and without limitation,display brightness, positioning of elements of display 109, font type,font size, text color, and what elements of display 109 are enabled. Insome embodiments, communications module 150 may be used to communicatewith one or more external devices such as headphones, which may be usedto output audible signals to a user including, for example and withoutlimitation, translations of analyzed speech. In certain embodiments,communications module 150 may be used to make and receive phone calls.

Where glasses with display 100 are used to receive incoming phone callsor function with communications module 150 to make outgoing phone calls,glasses with display 100 may, for example and without limitation,display an alert that a call is being received. Such an alert mayinclude text, an icon, the phone number associated with the incomingcall, or the name of the party associated with the incoming call.Glasses with display 100 may display text of the spoken words of theperson on the incoming call as described above with respect to otherspeakers. In some embodiments, glasses with display 100 may conduct theincoming or outgoing phone call by using user microphone array 115.

In some embodiments, one or more functions or parameters of glasses withdisplay 100 may be set by voice control. In such an embodiment,processor 123 may use audio signals 121 b from user microphone array 115and speech recognition as described above with respect to closedcaptioning operation to identify commands spoken by a user and to updateoperating modes and parameters of glasses with display 100 based on thereceived speech.

In some embodiments, as depicted in FIG. 1, glasses with display 100 mayinclude one or more external controls including, for example and withoutlimitation, on/off switch 161 and brightness switch 163. On/off switch161 may allow a user to enable or disable display 109 of glasses withdisplay 100 and to control other functions while glasses with display100 are in use. On/off switch 161 may also be used to answer an incomingphone call, make an outgoing phone call, or terminate a call. In otherembodiments, glasses with display 100 may be turned off by removing theglasses from the user's head and closing one or both arms or temples 105a, 105 b. Brightness switch 163 may be used to change the brightness ofdisplay 109.

The foregoing outlines features of several embodiments so that a personof ordinary skill in the art may better understand the aspects of thepresent disclosure. Such features may be replaced by any one of numerousequivalent alternatives, only some of which are disclosed herein. One ofordinary skill in the art should appreciate that they may readily usethe present disclosure as a basis for designing or modifying otherprocesses and structures for carrying out the same purposes and/orachieving the same advantages of the embodiments introduced herein. Oneof ordinary skill in the art should also realize that such equivalentconstructions do not depart from the spirit and scope of the presentdisclosure and that they may make various changes, substitutions, andalterations herein without departing from the spirit and scope of thepresent disclosure.

1. A method comprising: providing glasses with display, the glasses withdisplay including: a bridge; two temples hingedly coupled to the bridge;a directional microphone array, the directional microphone arrayincluding two or more microphones positioned on the bridge or thetemples; two lenses positioned in the bridge, at least one of the lensesincluding a display, the display visible by the user; and a processor;receiving sound from a first speaker with the directional microphonearray as an audio signal; recognizing speech from the audio signal withthe processor; determining directional information for a first speechorigination position of the first speaker based on differential analysisof the sound received by the two or more microphones in the directionalmicrophone array using the processor; outputting text speech output fromthe processor; displaying the text speech output with the display;outputting directional information from the processor; and displayingthe directional information with the display.
 2. The method of claim 1,wherein displaying the directional information with the displaycomprises displaying an arrow in the user's field of vision.
 3. Themethod of claim 2, further comprising rotating the arrow to point towardthe first speaker when the first speaker is speaking.
 4. The method ofclaim 3, wherein the method further comprises: determining directionalinformation for a second speech origination position of a second speakerbased on differential analysis of the sound received by the two or moremicrophones in the directional microphone array using the processor; androtating the arrow to point toward the second speaker when the secondspeaker is speaking.
 5. The method of claim 4, wherein displaying thedirectional information with the display comprises displaying the textbox in a position within the user's field of vision corresponding withthe directional information for a first speaker.
 6. The method of claim5, wherein the text box is displayed under or in the direction of thefirst speaker when the first speaker is speaking.
 7. The method of claim6, wherein the text box is displayed under or in the direction of thesecond speaker when the second speaker is speaking.
 8. A methodcomprising: providing glasses with display, the glasses with displayincluding: a bridge; two temples hingedly coupled to the bridge; adirectional microphone array, the directional microphone array includingtwo or more microphones positioned on the bridge or the temples; twolenses positioned in the bridge, at least one of the lenses including adisplay, the display visible by the user; a camera; a sensor module, thesensor module including one or more of an accelerometer, magnetometer,and gyro; and a processor; determining a position in which a user isfacing using the camera; detecting a speaker using the camera; and usingthe camera information from the sensor package to determine the positionof the speaker within the user's field of vision.
 9. The method of claim8 further comprising displaying a visual indicator on the display toidentify the speaker.
 10. The method of claim 9, wherein the visualindicator is a shape around a face of a speaker.
 11. The method of claim9 further comprising: determining the direction in which eyes of theuser are facing; and positioning elements of the display within thefield of view of the user.